feat: Add file loading code for flag overrides - #449

Summary

This is the first of a series that adds flag overrides, as defined by the OVERRIDE specification, to the Ruby server SDK. The existing FDv1 and FDv2 file data sources keep their current behavior: this PR does not change lib/ldclient-rb/impl/integrations/file_data_source.rb, file_data_source_v2.rb, or their specs, and the override feature is additive throughout the series.

The file-based override source must tolerate a missing file, a file that is being rewritten in place, and a burst of change notifications, and it must keep the last good data when a reload fails. This change adds LaunchDarkly::Impl::FileData, the file reading, parsing, merging, and reloading code that source needs:

• Document parsing. The format comes from the file name: a name that ends in .yaml or .yml, in any case (a file named .yaml included), is YAML, and any other file, including one with no extension, is JSON. The parser checks that the document and its flags, flagValues, and segments members are objects; a JSON null, array or scalar is an error.
  • Whether an empty file is valid depends on the format. An empty or white-space JSON file fails to parse (the file has no content, which is not a JSON document). Empty or white-space YAML (a tab included), or YAML that is only a comment, --- or a null, is a document with no entries.
  • Files are read as bytes and taken as UTF-8, without a UTF-8 byte order mark, whatever the process's default encoding. Content that is not valid UTF-8 fails to parse in either format; a file with a UTF-16 or UTF-32 byte order mark is reported as such (encoded in UTF-16 or UTF-32; data files must be UTF-8).
  • JSON is parsed with JSON.parse, so JSON numbers are numbers (1e3 is 1000.0), and content after the document, a # comment or a trailing comma is an error. Comments and duplicate keys inside one object follow the application's json gem: 3.x rejects both.
• An ordered merge of several documents with duplicate keys handling of fail or ignore, expansion of each flagValues entry into a flag that is on and serves the value by fallthrough (so an evaluation reports the FALLTHROUGH reason kind, as the specification describes), deserialization into the data model classes, and per-document entry counts. The key under which an entry appears is its key for every purpose: the item is stored under it, duplicates are detected by it, and the definition's own key is set to it, so evaluations report the flag by that key.
• A reloader that serializes reloads, debounces change signals, keeps the last good result when a reload fails, retries after a bounded delay, reports each distinct failure once, and can skip a reload whose paths, presence and content are identical to the last applied ones. A configured file that does not exist can be made to contribute no data. An exception raised by the consumer's apply is treated as a failed reload: nothing from that load is remembered, the failure is reported, and the reload is retried. An error that a load does not expect is logged and retried, and the worker thread survives it.
• In-place rewrites. Applications and editors truncate a file and then write the new content, and a load in between finds the file empty. A load that finds a file with no content (empty, white space, or YAML read as nothing) while the data in effect has content from that file holds the file back: it keeps the current data, logs at debug level, and arms the retry, without reporting a failure.
  • A file is held back until it has stayed so, unwritten, for one retry delay (1 second), timed from the clock read after that file's own read. A change of its state (modification time and size, read through any symbolic link with the poller's helper) is a new rewrite and starts the delay over, as does a read that finds it with content, even content that fails to parse, or missing.
  • Each file is held for at most five retry delays from its first hold; its bound starts over only at a load that holds nothing back, whether that load applies, is skipped, or fails.
  • A load reads and classifies every file, even after one fails to read or parse, so files emptied together are held and applied together; it then fails with the first failure, holds files back, or merges, in that order.
  • The retry of a hold is armed for the rest of the first due file's delay or bound, plus a tenth of the retry delay.
  • A file that contributed no content to the data in effect (a placeholder that has always been empty, a file that was missing, a file whose emptying has been applied) is parsed at once, and so is every file when retries are off.
  • After the delay, an empty JSON file fails to parse and keeps the data; an empty YAML file has no entries.
• A stat-based poller that compares existence, modification time, and size, through symbolic links. A file that appears or disappears is a change.
• A watcher that uses rb-inotify on Linux (a dependency of the optional listen gem) to watch the directory of each file without descending into subdirectories, and listen elsewhere. It picks up a file that does not exist yet when it appears, and it watches each directory on its own: a directory that does not exist is retried once per second while the others stay watched, and one change is signaled when a retry sets up a watch. Non-recursive watching matters because listen scans the whole tree under a watched directory and fails when a subdirectory cannot be read, as happens for a file in the system temporary directory. On Linux, a watched directory that is deleted is watched again once it exists, and one change is signaled so the reload picks up what was written meanwhile. A configured file that is a symbolic link is followed: the directory of the link's target is watched for the target's name, and the links are resolved again after every reported change, after a retry sets up watches, and when a directory that holds a target is deleted, as a Kubernetes ConfigMap update does. A link that cannot be resolved, for example because its target's directory does not exist yet or was deleted, is resolved again once per second until its target is watched, and one change is then signaled. A relative link is expanded against the real path of the directory that holds it, so a .. in it is right when a directory on the way is itself a link.

A new spec, spec/integrations/file_data_source_compatibility_spec.rb, pins the behaviors of the existing file data sources that their other specs did not assert (per-file version numbering, keying by the entry's own key, the on-plus-fallthrough expansion of flagValues, no retry of a failed load, log messages, modification-time-only polling, no reaction to a deleted file when polling, the FDv1 poller's reload cadence after a change, and use of listen when it is present), so that they stay as they are while this code exists beside them.

Not changed: YAML that is cut off mid-write can still parse, with fewer entries or a value cut short, so the override source docs recommend replacing a YAML file in one operation. A writer that pauses longer than the retry delay gets the empty file applied (YAML) or reported (JSON).

Testing

• Unit specs for the document parser (format by name, empty JSON and YAML, UTF-8 and byte order marks, JSON numbers and trailing content), merge, poller (including symbolic links), reloader (debounce, retry, last-good retention, sequential reloads, stop, and the hold of an emptied file), and watcher (including a directory with an unreadable subdirectory, the inotify thread lifecycle, symbolic links to files in the same and in another directory, a link that is replaced, a ConfigMap-style update, a relative link in a directory reached through another link, and a target directory that appears late or is recreated), the new compatibility pins, and the unchanged existing file data source specs.
• The reloader specs rewrite their files in place with real, deliberately slow writes (InPlaceWrites.rewrite in spec/in_place_writes.rb), and the hold's delays, bounds and retries are tested on a controlled clock with synchronous loads, with a few real-clock tests for the retry firing.
• The in-place rewrite handling was reviewed in eight rounds as fix: Keep overrides through an in-place rewrite of their files #479, with multi-agent and Bugbot reviews, and is shared with the other SDKs' ports. On that code: a mutation check reverted each part of the fix on its own (67 mutants); 60 fail at least one test, and the other 7 are equivalent (the retries-off guard, the clearing of held-file records on apply or skip, starting over every file instead of the files read, and the path or the presence alone in the digest). The changed spec files ran 10 times in a row on Ruby 3.2 and on JRuby 9.4 with no failures.
• Full suite and RuboCop are clean, apart from can be automatically loaded by Bundler.require, which needs write access to the system gem directory on the test machine. Each new spec was checked against a deliberate defect in the code it covers, and each compatibility pin against a deliberate change to the existing source it pins.

SDK-3249

_{Opened from termag}